Pose tracking pipeline

ABSTRACT

A method of tracking a subject includes receiving from a source a depth image of a scene including the subject. The depth image includes a depth for each of a plurality of pixels. The method further includes identifying pixels of the depth image that image the subject and deriving from the identified pixels of the depth image one or more machine readable data structures representing the subject as a model including a plurality of shapes.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/088,184 filed Nov. 22, 2013, which is a continuation of U.S. patentapplication Ser. No. 13/871,974 filed Apr. 26, 2013, now U.S. Pat. No.8,610,665, which is a continuation of U.S. patent application Ser. No.13/408,775 filed Feb. 29, 2012, now U.S. Pat. No. 8,553,939, which is acontinuation of U.S. patent application Ser. No. 12/603,437, filed Oct.21, 2009, now U.S. Pat. No. 8,295,546, which is a continuation-in-partof U.S. patent application Ser. No. 12/367,435, filed Feb. 6, 2009,which claims priority to U.S. Provisional Patent Application No.61/148,892, filed Jan. 30, 2009. The above applications are herebyincorporated herein by reference in their entirety.

BACKGROUND

Many computer games and other computer vision applications utilizecomplicated controls to allow users to manipulate game characters orother aspects of an application. Such controls can be difficult tolearn, thus creating a barrier to entry for many games or otherapplications. Furthermore, such controls may be very different from theactual game actions or other application actions for which they areused. For example, a game control that causes a game character to swinga baseball bat may not at all resemble the actual motion of swinging abaseball bat.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

A method of tracking a human subject is disclosed. The method includesreceiving from a source a depth image of a scene including the subject.The depth image includes a depth for each of a plurality of pixels. Themethod further includes identifying pixels of the depth image that imagethe subject and deriving from the identified pixels of the depth imageone or more machine readable data structures representing the subject asa model including a plurality of shapes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows an embodiment of an exemplary target recognition,analysis, and tracking system tracking a game player playing a boxinggame.

FIG. 1B shows the game player of FIG. 1A throwing a punch that istracked and interpreted as a game control that causes a player avatar tothrow a punch in game space.

FIG. 2 schematically shows a computing system in accordance with anembodiment of the present disclosure.

FIG. 3 shows an exemplary body model used to represent a human target.

FIG. 4 shows a substantially frontal view of an exemplary skeletal modelused to represent a human target.

FIG. 5 shows a skewed view of an exemplary skeletal model used torepresent a human target.

FIG. 6 shows a pipeline for tracking a target.

FIG. 7 shows a scene as viewed by a depth camera with schematic datastructures showing data used to track a target.

FIG. 8 schematically shows a progression of data through a target posetracking pipeline.

DETAILED DESCRIPTION

The present disclosure is directed to target recognition, analysis, andtracking. In particular, the use of a depth camera or other source foracquiring depth information for one or more targets is disclosed. Suchdepth information may then be used to efficiently and accurately modeland track the one or more targets, as described in detail below. Thetarget recognition, analysis, and tracking described herein provides arobust platform in which one or more targets can be consistently trackedat a relatively fast frame rate, even when the target(s) move into posesthat have been considered difficult to analyze using other approaches(e.g., when two or more targets partially overlap and/or occlude oneanother; when a portion of a target self-occludes another portion of thesame target, when a target changes its topographical appearance (e.g., ahuman touching his or her head), etc.).

FIG. 1A shows a nonlimiting example of a target tracking system 10. Inparticular, FIG. 1A shows a computer gaming system 12 that may be usedto play a variety of different games, play one or more different mediatypes, and/or control or manipulate non-game applications. FIG. 1A alsoshows a display 14 in the form of a high-definition television, or HDTV16, which may be used to present game visuals to game players, such asgame player 18. Furthermore, FIG. 1A shows a capture device in the formof a depth camera 20, which may be used to visually monitor one or moregame players, such as game player 18. The example shown in FIG. 1A isnonlimiting. As described below with reference to FIG. 2, a variety ofdifferent types of target tracking systems may be used without departingfrom the scope of this disclosure.

A target tracking system may be used to recognize, analyze, and/or trackone or more targets, such as game player 18. FIG. 1A shows a scenario inwhich game player 18 is tracked using depth camera 20 so that themovements of game player 18 may be interpreted by gaming system 12 ascontrols that can be used to affect the game being executed by gamingsystem 12. In other words, game player 18 may use his movements tocontrol the game. The movements of game player 18 may be interpreted asvirtually any type of game control.

The example scenario illustrated in FIG. 1A shows game player 18 playinga boxing game that is being executed by gaming system 12. The gamingsystem uses HDTV 16 to visually present a boxing opponent 22 to gameplayer 18. Furthermore, the gaming system uses HDTV 16 to visuallypresent a player avatar 24 that gaming player 18 controls with hismovements. As shown in FIG. 1B, game player 18 can throw a punch inphysical/world space as an instruction for player avatar 24 to throw apunch in game/virtual space. Gaming system 12 and depth camera 20 can beused to recognize and analyze the punch of game player 18 in physicalspace so that the punch can be interpreted as a game control that causesplayer avatar 24 to throw a punch in game space. For example, FIG. 1Bshows HDTV 16 visually presenting player avatar 24 throwing a punch thatstrikes boxing opponent 22 responsive to game player 18 throwing a punchin physical space.

Other movements by game player 18 may be interpreted as other controls,such as controls to bob, weave, shuffle, block, jab, or throw a varietyof different power punches. Furthermore, some movements may beinterpreted into controls that serve purposes other than controllingplayer avatar 24. For example, the player may use movements to end,pause, or save a game, select a level, view high scores, communicatewith a friend, etc.

In some embodiments, a target may include a human and an object. In suchembodiments, for example, a player of an electronic game may be holdingan object, such that the motions of the player and the object areutilized to adjust and/or control parameters of the electronic game. Forexample, the motion of a player holding a racket may be tracked andutilized for controlling an on-screen racket in an electronic sportsgame. In another example, the motion of a player holding an object maybe tracked and utilized for controlling an on-screen weapon in anelectronic combat game.

Target tracking systems may be used to interpret target movements asoperating system and/or application controls that are outside the realmof gaming. Virtually any controllable aspect of an operating systemand/or application, such as the boxing game shown in FIGS. 1A and 1B,may be controlled by movements of a target, such as game player 18. Theillustrated boxing scenario is provided as an example, but is not meantto be limiting in any way. To the contrary, the illustrated scenario isintended to demonstrate a general concept, which may be applied to avariety of different applications without departing from the scope ofthis disclosure.

The methods and processes described herein may be tied to a variety ofdifferent types of computing systems. FIGS. 1A and 1B show a nonlimitingexample in the form of gaming system 12, HDTV 16, and depth camera 20.As another, more general, example, FIG. 2 schematically shows acomputing system 40 that may perform one or more of the targetrecognition, tracking, and analysis methods and processes describedherein. Computing system 40 may take a variety of different forms,including, but not limited to, gaming consoles, personal computingsystems, public computing systems, human-interactive robots, militarytracking and/or targeting systems, and character acquisition systemsoffering green-screen or motion-capture functionality, among others.

Computing system 40 may include a logic subsystem 42, a data-holdingsubsystem 44, a display subsystem 46, and/or a capture device 48. Thecomputing system may optionally include components not shown in FIG. 2,and/or some components shown in FIG. 2 may be peripheral components thatare not integrated into the computing system.

Logic subsystem 42 may include one or more physical devices configuredto execute one or more instructions. For example, the logic subsystemmay be configured to execute one or more instructions that are part ofone or more programs, routines, objects, components, data structures, orother logical constructs. Such instructions may be implemented toperform a task, implement a data type, transform the state of one ormore devices, or otherwise arrive at a desired result. The logicsubsystem may include one or more processors that are configured toexecute software instructions. Additionally or alternatively, the logicsubsystem may include one or more hardware or firmware logic machinesconfigured to execute hardware or firmware instructions. The logicsubsystem may optionally include individual components that aredistributed throughout two or more devices, which may be remotelylocated in some embodiments.

Data-holding subsystem 44 may include one or more physical devicesconfigured to hold data and/or instructions executable by the logicsubsystem to implement the herein described methods and processes. Whensuch methods and processes are implemented, the state of data-holdingsubsystem 44 may be transformed (e.g., to hold different data).Data-holding subsystem 44 may include removable media and/or built-indevices. Data-holding subsystem 44 may include optical memory devices,semiconductor memory devices (e.g., RAM, EEPROM, flash, etc.), and/ormagnetic memory devices, among others. Data-holding subsystem 44 mayinclude devices with one or more of the following characteristics:volatile, nonvolatile, dynamic, static, read/write, read-only, randomaccess, sequential access, location addressable, file addressable, andcontent addressable. In some embodiments, logic subsystem 42 anddata-holding subsystem 44 may be integrated into one or more commondevices, such as an application specific integrated circuit or a systemon a chip.

FIG. 2 also shows an aspect of the data-holding subsystem in the form ofcomputer-readable removable media 50, which may be used to store and/ortransfer data and/or instructions executable to implement the hereindescribed methods and processes.

Display subsystem 46 may be used to present a visual representation ofdata held by data-holding subsystem 44. As the herein described methodsand processes change the data held by the data-holding subsystem, andthus transform the state of the data-holding subsystem, the state ofdisplay subsystem 46 may likewise be transformed to visually representchanges in the underlying data. As a nonlimiting example, the targetrecognition, tracking, and analysis described herein may be reflectedvia display subsystem 46 in the form of a game character that changesposes in game space responsive to the movements of a game player inphysical space. Display subsystem 46 may include one or more displaydevices utilizing virtually any type of technology. Such display devicesmay be combined with logic subsystem 42 and/or data-holding subsystem 44in a shared enclosure, or such display devices may be peripheral displaydevices, as shown in FIGS. 1A and 1B.

Computing system 40 further includes a capture device 48 configured toobtain depth images of one or more targets. Capture device 48 may beconfigured to capture video with depth information via any suitabletechnique (e.g., time-of-flight, structured light, stereo image, etc.).As such, capture device 48 may include a depth camera, a video camera,stereo cameras, and/or other suitable capture devices.

For example, in time-of-flight analysis, the capture device 48 may emitinfrared light to the target and may then use sensors to detect thebackscattered light from the surface of the target. In some cases,pulsed infrared light may be used, wherein the time between an outgoinglight pulse and a corresponding incoming light pulse may be measured andused to determine a physical distance from the capture device to aparticular location on the target. In some cases, the phase of theoutgoing light wave may be compared to the phase of the incoming lightwave to determine a phase shift, and the phase shift may be used todetermine a physical distance from the capture device to a particularlocation on the target.

In another example, time-of-flight analysis may be used to indirectlydetermine a physical distance from the capture device to a particularlocation on the target by analyzing the intensity of the reflected beamof light over time, via a technique such as shuttered light pulseimaging.

In another example, structured light analysis may be utilized by capturedevice 48 to capture depth information. In such an analysis, patternedlight (i.e., light displayed as a known pattern such as grid pattern, astripe pattern, a constellation of dots, etc.) may be projected onto thetarget. Upon striking the surface of the target, the pattern may becomedeformed, and this deformation of the pattern may be studied todetermine a physical distance from the capture device to a particularlocation on the target.

In another example, the capture device may include two or morephysically separated cameras that view a target from different angles toobtain visual stereo data. In such cases, the visual stereo data may beresolved to generate a depth image.

In other embodiments, capture device 48 may utilize other technologiesto measure and/or calculate depth values. Additionally, capture device48 may organize the calculated depth information into “Z layers,” i.e.,layers perpendicular to a Z axis extending from the depth camera alongits line of sight to the viewer.

In some embodiments, two or more different cameras may be incorporatedinto an integrated capture device. For example, a depth camera and avideo camera (e.g., RGB video camera) may be incorporated into a commoncapture device. In some embodiments, two or more separate capturedevices may be cooperatively used. For example, a depth camera and aseparate video camera may be used. When a video camera is used, it maybe used to provide target tracking data, confirmation data for errorcorrection of target tracking, image capture, face recognition,high-precision tracking of fingers (or other small features), lightsensing, and/or other functions.

It is to be understood that at least some target analysis and trackingoperations may be executed by a logic machine of one or more capturedevices. A capture device may include one or more onboard processingunits configured to perform one or more target analysis and/or trackingfunctions. A capture device may include firmware to facilitate updatingsuch onboard processing logic.

Computing system 40 may optionally include one or more input devices,such as controller 52 and controller 54. Input devices may be used tocontrol operation of the computing system. In the context of a game,input devices, such as controller 52 and/or controller 54 can be used tocontrol aspects of a game not controlled via the target recognition,tracking, and analysis methods and procedures described herein. In someembodiments, input devices such as controller 52 and/or controller 54may include one or more of accelerometers, gyroscopes, infraredtarget/sensor systems, etc., which may be used to measure movement ofthe controllers in physical space. In some embodiments, the computingsystem may optionally include and/or utilize input gloves, keyboards,mice, track pads, trackballs, touch screens, buttons, switches, dials,and/or other input devices. As will be appreciated, target recognition,tracking, and analysis may be used to control or augment aspects of agame, or other application, conventionally controlled by an inputdevice, such as a game controller. In some embodiments, the targettracking described herein can be used as a complete replacement to otherforms of user input, while in other embodiments such target tracking canbe used to complement one or more other forms of user input.

Computing system 40 may be configured to perform the target trackingmethods described herein. However, it should be understood thatcomputing system 40 is provided as a nonlimiting example of a devicethat may perform such target tracking. Other devices are within thescope of this disclosure.

Computing system 40, or another suitable device, may be configured torepresent each target with a model. As described in more detail below,information derived from such a model can be compared to informationobtained from a capture device, such as a depth camera, so that thefundamental proportions or shape of the model, as well as its currentpose, can be adjusted to more accurately represent the modeled target.The model may be represented by one or more polygonal meshes, by a setof mathematical primitives, and/or via other suitable machinerepresentations of the modeled target.

FIG. 3 shows a nonlimiting visual representation of an example bodymodel 70. Body model 70 is a machine representation of a modeled target(e.g., game player 18 from FIGS. 1A and 1B). The body model may includeone or more data structures that include a set of variables thatcollectively define the modeled target in the language of a game orother application/operating system.

A model of a target can be variously configured without departing fromthe scope of this disclosure. In some examples, a model may include oneor more data structures that represent a target as a three-dimensionalmodel comprising rigid and/or deformable shapes, or body parts. Eachbody part may be characterized as a mathematical primitive, examples ofwhich include, but are not limited to, spheres, anisotropically-scaledspheres, cylinders, anisotropic cylinders, smooth cylinders, boxes,beveled boxes, prisms, and the like.

For example, body model 70 of FIG. 3 includes body parts bp1 throughbp14, each of which represents a different portion of the modeledtarget. Each body part is a three-dimensional shape. For example, bp3 isa rectangular prism that represents the left hand of a modeled target,and bp5 is an octagonal prism that represents the left upper-arm of themodeled target. Body model 70 is exemplary in that a body model maycontain any number of body parts, each of which may be anymachine-understandable representation of the corresponding part of themodeled target.

A model including two or more body parts may also include one or morejoints. Each joint may allow one or more body parts to move relative toone or more other body parts. For example, a model representing a humantarget may include a plurality of rigid and/or deformable body parts,wherein some body parts may represent a corresponding anatomical bodypart of the human target. Further, each body part of the model maycomprise one or more structural members (i.e., “bones” or skeletalparts), with joints located at the intersection of adjacent bones. It isto be understood that some bones may correspond to anatomical bones in ahuman target and/or some bones may not have corresponding anatomicalbones in the human target.

The bones and joints may collectively make up a skeletal model, whichmay be a constituent element of the body model. In some embodiments, askeletal model may be used instead of another type of model, such asmodel 70 of FIG. 3. The skeletal model may include one or more skeletalmembers for each body part and a joint between adjacent skeletalmembers. Exemplary skeletal model 80 and exemplary skeletal model 82 areshown in FIGS. 4 and 5, respectively. FIG. 4 shows a skeletal model 80as viewed from the front, with joints j1 through j33. FIG. 5 shows askeletal model 82 as viewed from a skewed view, also with joints j1through j33.

Skeletal model 82 further includes roll joints j34 through j47, whereeach roll joint may be utilized to track axial roll angles. For example,an axial roll angle may be used to define a rotational orientation of alimb relative to its parent limb and/or the torso. For example, if askeletal model is illustrating an axial rotation of an arm, roll jointj40 may be used to indicate the direction the associated wrist ispointing (e.g., palm facing up). By examining an orientation of a limbrelative to its parent limb and/or the torso, an axial roll angle may bedetermined. For example, if examining a lower leg, the orientation ofthe lower leg relative to the associated upper leg and hips may beexamined in order to determine an axial roll angle.

A skeletal model may include more or fewer joints without departing fromthe spirit of this disclosure.

As described above, some models may include a skeleton and/or other bodyparts that serve as a machine representation of a modeled target. Insome embodiments, a model may alternatively or additionally include awireframe mesh, which may include hierarchies of rigid polygonal meshes,one or more deformable meshes, or any combination of the two.

The above described body part models and skeletal models are nonlimitingexample types of models that may be used as machine representations of amodeled target. Other models are also within the scope of thisdisclosure. For example, some models may include polygonal meshes,patches, non-uniform rational B-splines, subdivision surfaces, or otherhigh-order surfaces. A model may also include surface textures and/orother information to more accurately represent clothing, hair, and/orother aspects of a modeled target. A model may optionally includeinformation pertaining to a current pose, one or more past poses, and/ormodel physics. It is to be understood that a variety of different modelsthat can be posed are compatible with the herein described targetrecognition, analysis, and tracking.

As mentioned above, a model serves as a representation of a target, suchas game player 18 in FIGS. 1A and 1B. As the target moves in physicalspace, information from a capture device, such as depth camera 20 inFIGS. 1A and 1B, can be used to adjust a pose and/or the fundamentalsize/shape of the model so that it more accurately represents thetarget.

FIG. 6 shows a flow diagram of an example pipeline 100 for tracking oneor more targets. In some embodiments, the target(s) may be human gameplayers. Pipeline 100 may be executed by a computing system (e.g.,gaming system 12 shown in FIG. 1A and/or computing system 40 shown inFIG. 2) to track one or more players interacting with an electronicgame. As introduced above, tracking of the players allows physicalmovements of those players to act as real-time user controls that adjustand/or control parameters of the electronic game. It is to be understoodthat gaming is provided as a nonlimiting example, and the disclosedpipeline may be used to track human or nonhuman targets for a variety ofother purposes.

The disclosed pipeline can be used to accurately and efficiently trackone or more humans that are present in the field of view of a depthcamera. The pipeline can model and track one or more humans in realtime, thus providing a responsive, immersive, and realistic experiencefor a human being tracked. Furthermore, the pipeline is believed to beefficient, so as to limit the computing resources used to execute thepipeline.

Pipeline 100 includes six conceptual phases: depth image acquisition102, background removal 104, foreground pixel assignment 106, modelfitting 108, model resolution 110, and reporting 112.

Depth image acquisition 102 may include receiving an observed depthimage of the target from a source. In some embodiments, the source maybe a depth camera configured to obtain depth information about thetarget via a suitable technique such as time-of-flight analysis,structured light analysis, stereo vision analysis, or other suitabletechniques. The observed depth image may include a plurality of observedpixels, where each observed pixel has an observed depth value. Theobserved depth value includes depth information of the target as viewedfrom the source.

The depth image may optionally be represented as a pixel matrix thatincludes, for each pixel address, a depth value indicating a world spacedepth from the plane of the depth camera, or another suitable referenceplane, to a surface at that pixel address.

FIG. 7 schematically shows a scene 150 captured by a depth camera. Thedepth camera determines a Z-value of a surface at each pixel address. Asan example, FIG. 7 schematically shows a data structure 152 used torepresent pixel 154 at pixel address [1436, 502]. Data structure 152 maybe an element of a pixel matrix, for example. Data structure 152includes a Z-value of 425 for pixel 154, thus indicating that thesurface at that pixel address, in this case a wall, is 425 units deep inworld space. As another example, a data structure 156 is used torepresent pixel 158 at pixel address [913, 693]. Data structure 156includes a Z-value of 398 for pixel 158, thus indicating that thesurface at that pixel address, in this case a door, is 398 units deep inworld space. As another example, a data structure 160 is used torepresent pixel 162 at pixel address [611, 597]. Data structure 160includes a Z-value of 173 for pixel 162, thus indicating that thesurface at that pixel address, in this case a human target, is 173 unitsdeep in world space. While three pixels are provided as examples above,it is to be understood that every pixel captured by a capture device, ora downsampled set thereof, may be represented in this manner.

As shown at 114 of FIG. 6, depth image acquisition 102 may optionallyinclude downsampling the observed depth image to a lower processingresolution. Downsampling to a lower processing resolution may allow theobserved depth image to be more easily utilized and/or more quicklyprocessed with less computing overhead.

As shown at 116 of FIG. 6, depth image acquisition 102 may optionallyinclude removing and/or smoothing one or more high-variance and/or noisydepth values from the observed depth image. Such high-variance and/ornoisy depth values in the observed depth image may result from a numberof different sources, such as random and/or systematic errors occurringduring the image capturing process, defects and/or aberrations resultingfrom the capture device, etc. Since such high-variance and/or noisydepth values may be artifacts of the image capturing process, includingthese values in any future analysis of the image may skew results and/orslow calculations. Thus, removal of such values may provide better dataintegrity and/or speed for future calculations.

Background removal 104 may include distinguishing targets that are to betracked from non-target background elements in the observed depth image.As used herein, the term “background” is used to describe anything inthe scene that is not part of the target(s) to be tracked. Thebackground may include elements that are in front of (i.e., closer tothe depth camera) than the target(s) to be tracked. Distinguishingforeground elements that are to be tracked from background elements thatmay be ignored can increase tracking efficiency and/or simplifydownstream processing.

Background removal 104 may include assigning each data point (e.g.,pixel) of the processed depth image a player index that identifies thatdata point as belonging to a particular target or to a non-targetbackground element. When such an approach is used, pixels or other datapoints assigned a background index can be removed from consideration inone or more subsequent phases of pipeline 100.

As an example, pixels corresponding to a first player can be assigned aplayer index equal to one, pixels corresponding to a second player canbe assigned a player index equal to two, and pixels that do notcorrespond to a target player can be assigned a player index equal tozero. Such player indices can be saved in any suitable manner. In someembodiments, a pixel matrix may include, at each pixel address, a playerindex indicating if a surface at that pixel address belongs to abackground element, a first player, a second player, etc. For example,FIG. 7 shows data structure 152 including a player index equal to zerofor wall pixel 154, data structure 156 including a player index equal tozero for door pixel 158, and data structure 160 including a player indexequal to one for player pixel 162. While this example shows theplayer/background indices as part of the same data structure that holdsthe depth values, other arrangements are possible. In some embodiments,depth information, player/background indices, body part indices, bodypart probability distributions, and other information may be tracked ina common data structure, such as a matrix addressable by pixel address.In other embodiments, different masks may be used to track informationthrough pipeline 100. The player index may be a discrete index or afuzzy index indicating a probability that a pixel belongs to aparticular target and/or the background.

A variety of different background removal techniques may be used. Somebackground removal techniques may use information from one or moreprevious frames to assist and improve the quality of background removal.For example, a depth history image can be derived from two or moreframes of depth information, where the depth value for each pixel is setto the deepest depth value that pixel experiences during the sampleframes. A depth history image may be used to identify moving objects inthe foreground of a scene (e.g., a human game player) from the nonmovingbackground elements. In a given frame, the moving foreground pixels arelikely to have depth values that are smaller than the correspondingdepth values (at the same pixel addresses) in the depth history image.In a given frame, the nonmoving background pixels are likely to havedepth values that match the corresponding depth values in the depthhistory image.

As one nonlimiting example, a connected island background removal may beused. Using a connected island approach, an input depth stream can beused to generate a set of samples (e.g., voxels) that can beconceptually unprojected back into world space. Foreground objects arethen isolated from background objects using information from previousframes. In particular, the process can be used to determine whether oneor more voxels in the grid are associated with a background bydetermining whether an object of the one or more objects in the grid ismoving. This may be accomplished, at least in part, by determiningwhether a given voxel is close to or behind a reference plate that is ahistory of the minimum or maximum values observed for backgroundobjects. The output from this process can be used to assign each datapoint (e.g., pixel) a player index or a background index. Connectedisland background removal is described in U.S. patent application Ser.No. 12/575,363, filed Oct. 7, 2009, the entirety of which is herebyincorporated herein by reference.

Additional or alternative background removal techniques can be used toassign each data point a player index or a background index, orotherwise distinguish foreground targets from background elements. Insome embodiments, particular portions of a background may be identified.For example, at 118 of FIG. 6, a floor in a scene may be identified aspart of the background. In addition to being removed from considerationwhen processing foreground targets, a found floor can be used as areference surface that can be used to accurately position virtualobjects in game space, stop a flood-fill that is part of generating aconnected island, and/or reject an island if its center is too close tothe floor plane.

A variety of different floor finding techniques may be used. In someembodiments, a depth image can be analyzed in screen space row by row.For selected candidate rows of the screen space depth image (e.g., rowsnear the bottom of the image), a straight depth line can be interpolatedthrough two candidate points that are believed to be located on a floorsurface. Boundary lines can then be fit to endpoints of the straightdepth lines. The boundary lines can be averaged and used to define aplane that is believed to correspond to the floor surface. Screen spacefloor detection of this kind is described in U.S. patent applicationSer. No. 12/563,456, filed Sep. 21, 2009, the entirety of which ishereby incorporated herein by reference.

In other embodiments, a floor finding technique may use three pointsfrom a depth image to define a candidate floor surface. The three pointsused to define the candidate can be randomly selected from a lowerportion of the depth image, for example. If the normal of the candidateis substantially vertical in world space, the candidate is considered,and if the normal of the candidate is not substantially vertical, thecandidate can be rejected. A candidate with a substantially verticalnormal can be scored by counting how many points from the depth imageare located below the candidate and/or what the average distance suchpoints are below the candidate. If the number of points below thecandidate exceeds a threshold and/or the average distance of pointsbelow the candidate exceeds a threshold, the candidate can be rejected.Different candidates are tested, and the candidate with the best scoreis saved. The saved candidate may be blessed as the actual floor if apredetermined number of candidates with lower scores are tested againstthe saved candidate.

Additional or alternative background removal techniques can be used toassign each data point a player index or a background index, orotherwise distinguish foreground targets from background elements. Forexample, in FIG. 6 pipeline 100 includes bad body rejection 120. In someembodiments, objects that are initially identified as foreground objectscan be rejected because they do not resemble any known target. Forexample, an object that is initially identified as a foreground objectcan be tested for basic criteria that are to be present in any objectsto be tracked (e.g., head and/or torso identifiable, bone lengths withinpredetermined tolerances, etc.). If an object that is initiallyidentified as being a candidate foreground object fails such testing, itmay be reclassified as a background element and/or subjected to furthertesting. In this way, moving objects that are not to be tracked, such asa chair pushed into the scene, can be classified as background elementsbecause such elements do not resemble a human target.

After foreground pixels are distinguished from background pixels,pipeline 100 further classifies the pixels that are considered tocorrespond to the foreground objects that are to be tracked. Inparticular, at foreground pixel assignment 106 of FIG. 6, eachforeground pixel is analyzed to determine what part of a player target'sbody that foreground pixel is likely to belong.

A variety of different foreground pixel assignment techniques can beused to assess which part of a player target's body, or a machinerepresentation of the body, a particular pixel is likely to belong. Apixel matrix or other data structure may include, for each pixeladdress, a body part index, confidence value, and/or body partprobability distribution indicating the part, or parts, to which a pixelis likely to belong. For example, FIG. 7 schematically shows datastructure 160 including a body part index equal to nine, whichcorresponds to an upper, right arm, for player pixel 162. In thesimplified version of FIG. 7, the body part index is resolved to asingle candidate body part (i.e., body part nine). In practice, the bodypart information may be a soft labeling that is represented as ahistogram over possible body parts for each pixel. In other words, aprobability distribution of all likely body parts may be used in someembodiments, as described in more detail below.

As one nonlimiting example, machine-learning can be used to assign eachforeground pixel a body part index and/or body part probabilitydistribution. The machine-learning approach analyzes a foreground objectusing information learned from analyzing a prior-trained collection ofknown poses. This approach can be used to assign each foreground pixel abody part index or distribution without any prior context (i.e.,knowledge of the prior frame is not needed).

In some embodiments, the machine-learning foreground pixel assignmentmay utilize one or more decision trees to analyze each foreground pixelof interest in an observed depth image. Such analysis can find abest-guess of the body part for that pixel and the confidence that thebest-guess is correct. In some embodiments, the best-guess may include aprobability distribution over two or more possible body parts, and theconfidence may be represented by the relative probabilities of thedifferent possible body parts.

At each node of the decision tree, an observed depth value comparisonbetween two pixels is made, and, depending on the result of thecomparison, a subsequent depth value comparison between two pixels ismade at the child node of the decision tree. The result of suchcomparisons at each node determines the pixels that are to be comparedat the next node. The terminal nodes of each decision tree result in abody part classification and associated confidence in theclassification.

In some embodiments, subsequent decision trees may be used toiteratively refine the best-guess of the body part for each pixel andthe confidence that the best-guess is correct. For example, once thepixels have been classified with the first classifier tree (based onneighboring depth values), a refining classification may be performed toclassify each pixel by using a second decision tree that looks at theprevious classified pixels and/or depth values. A third pass may also beused to further refine the classification of the current pixel bylooking at the previous classified pixels and/or depth values. It is tobe understood that virtually any number of iterations may be performed,with fewer iterations resulting in less computational expense and moreiterations potentially offering more accurate classifications and/orconfidences.

The decision trees may be constructed during a training mode in which asample of known models in known poses are analyzed to determine thequestions (i.e., tests) that can be asked at each node of the decisiontrees in order to produce accurate pixel classifications.

Turning back to FIG. 6, after foreground pixels are labeled with bodypart information, pipeline 100 includes model fitting 108, which findsone or more possible skeletons that serve as machine representations ofthe player target.

A variety of different model fitting techniques may be used. Duringmodel fitting 108, a human target is modeled as a skeleton including aplurality of skeletal points, each skeletal point having athree-dimensional location in world space. The various skeletal pointsmay correspond to actual joints of a human target, terminal ends of ahuman target's extremities, and/or points without a direct anatomicallink to the human target. Each skeletal point has at least three degreesof freedom (e.g., world space x, y, z). As such, the skeleton can befully defined by 3×λ values, where λ is equal to the total number ofskeletal points included in the skeleton. A skeleton with 31 skeletalpoints can be defined by 93 values, for example. As described withreference to FIG. 5 above, some skeletal points may account for axialroll angles.

The various model fitting approaches compatible with pipeline 100 mayuse depth information, background information, body part information,and/or prior trained anatomical and kinetic information to deduce one ormore skeleton(s) that closely model a human target.

As an example, the body part information that is assessed for theforeground pixels may be used to find one or more candidate locations(e.g., centroids) for one or more skeletal bones. Furthermore, aplurality of plausible skeletons may be assembled to include skeletalbones at different combinations of the plurality of candidate locations.The various plausible skeletons may then be scored, and the scoredproposals can be combined into a final estimate.

Clumps of foreground pixels may individually include body partprobability distributions indicating that a particular body part isprobable for that clump. In some cases, two or more clumps that arespaced apart from one another may indicate that the same body part isprobable. For example, the clumps of pixels actually showing the rightand left hands of a target may both be labeled with body partinformation indicating a high probability for a right hand body part. Assuch, two or more centroid candidates may be calculated for each bodypart. Each centroid candidate for a parituclar body part may berepresented in four dimensions—x, y, z and probability w that thecandidate belongs to that body part. In other words, each centroiddefines a location of a clump of neighboring foreground pixelsindividually having body part probability distributions indicating thatthat body part is probable for that clump of neighboring foregroundpixels. Furthermore, each centroid defines a single probabilityrepresenting all individual body part probability distributions withinthe clump.

Two or more different centroid candidates can be considered in finding askeleton that closely models a human target. The various candidatecentroids can be scored against one another (e.g., number of pixels inclump multiplied by average probability that pixels in the clump belongto a particular body part). The scores may be adjusted based on one ormore constraints (e.g., apply a penalty when a distance between thehighest scoring clump for a body part in a previous frame exceeds athreshold distance to the highest scoring clump for the same body partin the current frame). The scored centroids may be used to construct oneor more plausible skeletons, from which a single skeleton can bederived.

Considering plural plausible skeletons employs a probabilistic principleof least commitment to deal with uncertainty. As such, manypossibilities are considered throughout the model fitting phase of thepipeline, without making hard decisions until such decisions can nolonger be avoided.

When the data is unambiguous, the plurality of plausible skeletons willbe very similar to each other. If there are situations in which thereare numerous possibilities for one or more parts of the skeleton, thesample set will be more diverse, thus capturing the uncertainty.

The accuracy of the approximation can improve as the number of plausibleskeletons, n, increases. However, computational cost also increases as nincreases. The model fitting phase of the pipeline can be restrained tofocus on a relatively small number of samples (e.g., n<100). To improveresults when working with fewer samples, information from foregroundpixel assignment 106 and past skeletal motion information may be used toimprove the intelligence of the search for a set of proposal skeletons.The plausible skeletons can be evaluated against the input data, andother sources of information, to arrive at a final approximation.

As shown in FIG. 6, model fitting 108 may receive input from previousphases of pipeline 100. Model fitting 108 may receive one or more rawdepth images from depth image acquisition 102, player/backgroundinformation from background removal 104, and body part information fromforeground pixel assignment 106.

With all available inputs, foreground regions of the depth image may besegmented into a set of patches, which are regions of roughly consistentdepth. This effectively approximates the full depth image by a set ofsmall planar regions. For a modest loss of fidelity, this can reduce thebandwidth requirements from millions of pixel accesses, to thousands ofpatch accesses.

Plausible skeletons may then be proposed from the previous foregroundpixel assignments. The purpose of this phase is to convert pixel-wisebody part probability distributions into proposals for full skeletons(e.g., 93 values for a λ=31 skeleton). In the spirit of the principle ofleast commitment, all likely locations for a body part are considereduntil global information can be brought to bear. Therefore, this phasemay include two components: a body part proposer, which extractscandidate locations from foreground pixel assignment 106 for each bodypart independently (e.g., finding candidate centroids for each bodypart, as introduced above); and a skeleton generator, which combinesthese candidates into complete skeletons.

As discussed above, in at least some embodiments, model fitting can beused to find a plurality of different plausible skeletons. A proposedskeleton can be scored using a variety of different metrics. In FIG. 6,pipeline 100 includes model resolution 110, in which a single skeletonis derived from the plurality of plausible skeletons. A variety ofdifferent model resolution techniques may be used. In some embodiments,two or more plausible skeletons may be scored against each other basedon weight, observed motion over time, anticipated bone length,foreground/background crossing, and/or other factors. A proposedskeleton with a highest score may be selected; or the best scoringportions of two or more different proposed skeletons, from one or moredifferent frames, may be combined into a selected skeleton. Furthermore,various constraints (e.g., bone length, joint angle, collision testing,etc.) may be applied to one or more skeletons to shift the proposedskeleton(s) into a better matching pose.

Pipeline 100 also includes reporting 112, where the selected skeleton isreported for use by other applications. Reporting can be performed inany suitable manner. As a nonlimiting example, an applicationprogramming interface (API) may be used to report the selected skeleton.Such an (API) may be configured to communicate the joint positions,joint velocities, joint accelerations, confidences in positions,velocities, and/or accelerations, and/or other information related tothe selected skeleton for one or more targets. A content receiver (e.g.,a gaming application) may then use the reported information as desired.

FIG. 8 graphically shows a progression of data through a pose trackingpipeline. On the left, a scene 170 includes a human target 172. At 174,scene 170 is imaged by a depth camera and background information isremoved from the scene. Silhouette 176 schematically represents one ormore data structures capturing the depth information and player indexinformation (e.g., as captured during depth image acquisition 102 andbackground removal 104 of pipeline 100).

At 178, the depth information believed to belong to the player target isanalyzed to determine what part of a player target's body each pixel islikely to belong. Map 180 schematically represents one or more datastructures capturing the body part information (e.g., as deduced duringforeground pixel assignment 106 of pipeline 100).

At 182, a set of plausible skeletons are proposed, where each skeletonis an attempt to model the human target as a machine representation.Skeleton set 184 schematically represents one or more data structuresdefining the proposed skeletons (e.g., as proposed during model fitting108 of pipeline 100). It is to be understood that the graphicaldepictions of skeletons in FIGS. 4, 5, and 8 is nonlimiting. Skeletonswith a different number and/or configuration of skeletal points andskeletal bones may be used.

At 186, a skeleton is selected based on the set of plausible skeletons.Skeleton 188 schematically represents one or more data structuresdefining the selected skeleton (e.g., as selected during modelresolution 110 of pipeline 100).

At 190, the selected skeleton is reported (e.g., as described withreference to model reporting 112 of pipeline 100). As indicated at 192,the reported skeleton may be used as an input by an operating system,one or more applications, or any other suitable receiver.

It is to be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated may beperformed in the sequence illustrated, in other sequences, in parallel,or in some cases omitted. Likewise, the order of the above-describedprocesses may be changed.

The subject matter of the present disclosure includes all novel andnonobvious combinations and subcombinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

The invention claimed is:
 1. A method performed using a system,comprising: based on at least one image of at least part of a sceneobtained from an image capture device, the at least part of the sceneincluding one or more parts of a human body in a first position,separating image data included in the at least one image andcorresponding to the one or more parts of the human body from image dataincluded in the at least one image and corresponding to a background ofthe at least part of the scene; developing a first model correspondingto the one or more parts of the human body based at least in part on theimage data corresponding to the one or more parts of the human body, thefirst model including one or more skeletal points; developing a secondmodel based at least in part on data associated with the one or moreskeletal points and including shapes corresponding to the one or moreparts of the human body; displaying the second model; and controllingoperation of a program executing on the system based at least in part onusing the data associated with the one or more skeletal points of thefirst model to detect movement of the one or more parts of the humanbody, the based at least in part on using the data associated with theone or more skeletal points of the first model to detect movement of theone or more parts of the human body performed by the system being basedat least in part on the at least one image of the at least part of thescene and an additional one or more images.
 2. The method of claim 1,wherein the shapes include a cylinder or a sphere.
 3. The method ofclaim 1, wherein the one or more skeletal points include a roll joint,each roll joint utilized to track an axial roll angle.
 4. The method ofclaim 1, wherein the program is a gaming program and the system is agaming system.
 5. The method of claim 1, wherein the image capturedevice includes a depth camera, and the at least one image includes atleast one depth image.
 6. The method of claim 5, further comprising:downsampling the at least one depth image to a downsampled at least onedepth image having a lower processing resolution than the at least onedepth image.
 7. The method of claim 5, wherein separating image dataincludes distinguishing one or more parts of the human body that are tobe tracked from background elements in the at least one depth image. 8.The method of claim 7, wherein the at least one depth image includes twoor more depth frames, and the method further comprises: deriving a depthhistory image from the two or more depth frames, wherein a depth valuefor each pixel of the depth history image is set to a deepest depthvalue the pixel experiences during the two or more depth frames; andwherein distinguishing one or more parts of the human body includesidentifying moving objects in a foreground of the scene from non-movingbackground elements using the depth history image.
 9. The method ofclaim 8, wherein identifying moving objects includes determining whetheran object is behind a reference plate that is a history of minimum ormaximum depth values observed for the non-moving background elements.10. The method of claim 1, wherein the one or more parts of the humanbody belong to a player of an electronic game holding an object, suchthat motions of the player and the object are utilized to adjust and/orcontrol parameters of the electronic game.
 11. The method of claim 1,wherein controlling operation of the program includes one or more ofending a game, pausing the game, selecting a level of the game, viewingscores associated with the game, or saving the game.
 12. A systemcomprising: a capture device configured to obtain one or more images ofat least part of a scene; a logic device; a data-holding deviceconfigured to hold instructions executable by the logic device to:obtain via the capture device at least one image of the at least part ofthe scene, the scene including one or more parts of a human body in afirst position; separate image data included in the at least one imageand corresponding to the one or more parts of the human body from imagedata included in the at least one image and corresponding to abackground of the at least part of the scene; develop a first modelcorresponding to the one or more parts of the human body using the imagedata corresponding to the one or more parts of the human body, the firstmodel including one or more skeletal points; develop a second modelbased at least in part on data associated with the one or more skeletalpoints and including shapes corresponding to the one or more parts ofthe human body; display the second model; and control operation of aprogram executing on the system based at least in part on using the dataassociated with the one or more skeletal points of the first model todetect movement of the one or more parts of the human body, the based atleast in part on using the data associated with the one or more skeletalpoints of the first model to detect movement of the one or more parts ofthe human body performed by the system being based at least in part onthe at least one image of the at least part of the scene and anadditional one or more images.
 13. A method performed using a system,comprising: based at least in part on a first set of images of at leastpart of a scene including one or more parts of a human body in a firstposition and obtained from a plurality of image capture devices, witheach of the plurality of image capture devices located in a differentposition relative to the other of the plurality of image capturedevices, developing a first model corresponding to the one or more partsof the human body in the first position, the first model including oneor more skeletal points; developing a first display model based at leastin part on data associated with the one or more skeletal points of thefirst model and including shapes corresponding to the one or more partsof the human body; displaying the first display model; developing asecond model based at least in part on a second set of images of atleast part of the scene including the one or more parts of the humanbody in a second position and obtained from the plurality of imagecapture devices, the second model including the one or more skeletalpoints developing a second display model based at least in part on dataassociated with the one or more skeletal points of the second model andincluding shapes corresponding to the one or more parts of the humanbody; displaying the second display model; determining differencesbetween the first model and the second model; and controlling at least aportion of operation of a program executing on the system based at leastin part on the differences between the first model and the second model.14. The method of claim 13, wherein the shapes include a cylinder or asphere.
 15. The method of claim 13, wherein the one or more body partsinclude a human finger.
 16. The method of claim 13, wherein the one ormore skeletal points include one or more joints, the one or more jointsinclude roll joints, each roll joint utilized to track axial rollangles.
 17. The method of claim 16, wherein the first model and thesecond model further include one or more model parts corresponding to anobject held by a player of a gaming program, and wherein motions of theplayer and the object are utilized to adjust and/or control parametersof the gaming program.
 18. The method of claim 16, wherein controllingoperation of the program includes one or more of ending a game, pausingthe game, selecting a level of the game, viewing scores associated withthe game, or saving the game.
 19. The method of claim 13, wherein theplurality of image capture devices includes a plurality of depthcameras, wherein the first set of images includes a first set of depthimages, and wherein the second set of images includes a second set ofdepth images.
 20. The method of claim 19, wherein forming the firstmodel includes separating image data included in the first set of depthimages and corresponding to the one or more parts of the human body fromimage data included in the first set of images and corresponding to abackground of the scene, and wherein forming the second model includesseparating image data included in the second set of images andcorresponding to the one or more parts of the human body from image dataincluded in the second set of image depth images and corresponding tothe background of the scene.
 21. The method of claim 13, wherein thesecond model is formed based at least in part on the first model andchanges in position between the one or more parts of the human body inthe first position in the first set of images and the one or more partsof the human body in the second position in the second set of images.22. A system comprising: a capture device configured to obtain one ormore images of at least part of a scene; a logic device; a data-holdingdevice configured to hold instructions executable by the logic deviceto: obtain via the capture device a first set of images of the at leastpart of the scene, the scene including one or more parts of a human bodyin a first position; develop a first model corresponding to the one ormore parts of the human body in the first position based at least inpart on the first set of images, the first model including one or moreskeletal points; and develop a first display model based at least inpart on data associated with the one or more skeletal points of thefirst model and including shapes corresponding to the one or more partsof the human body; display the first display model; develop a secondmodel based at least in part on a second set of images of at least partof the scene including the one or more parts of the human body in asecond position, the second model including the one or more skeletalpoints; develop a second display model based at least in part on dataassociated with the one or more skeletal points of the second model andincluding shapes corresponding to the one or more parts of the humanbody; display the second display model; determine differences betweenthe first model and the second model; and control operation of a programexecuting on the system based at least in part on differences betweenthe first model and the second model.